from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
reporting = HpMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()
We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup
time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.
KNeighborsClassifier_brute_force¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: algorithm=brute.
| estimator | function | n_samples_train | n_samples | n_features | accuracy_score_sklearn | accuracy_score_onnx | diff_accuracy_scores | |
|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.663 | 1.000 | 0.337 |
| 2 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 1.000 | 0.757 | 0.243 |
| 4 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.757 | 0.882 | 0.125 |
| 7 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.882 | 1.000 | 0.118 |
| 8 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 1.000 | 0.757 | 0.243 |
| 10 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.882 | 0.663 | 0.219 |
| 13 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.757 | 1.000 | 0.243 |
| 14 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 1.000 | 0.922 | 0.078 |
| 16 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 0.663 | 0.929 | 0.266 |
| 19 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 0.896 | 1.000 | 0.104 |
| 20 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 2 | 1.000 | 0.922 | 0.078 |
| 22 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 0.922 | 0.896 | 0.026 |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.011 | 0.000 | 6.961 | 0.0 | -1 | 1 | 18.648 | 0.044 | 0.663 | 0.001 | 0.001 | See | See |
| 3 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.011 | 0.000 | 7.000 | 0.0 | -1 | 5 | 0.319 | 0.005 | 1.000 | 0.036 | 0.036 | See | See |
| 6 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.012 | 0.000 | 6.851 | 0.0 | 1 | 100 | 18.714 | 0.074 | 0.882 | 0.001 | 0.001 | See | See |
| 9 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.012 | 0.001 | 6.549 | 0.0 | -1 | 100 | 0.318 | 0.006 | 1.000 | 0.038 | 0.038 | See | See |
| 12 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.013 | 0.000 | 6.158 | 0.0 | 1 | 5 | 3.703 | 0.020 | 0.896 | 0.004 | 0.004 | See | See |
| 15 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 100 | 0.011 | 0.000 | 7.017 | 0.0 | 1 | 1 | 0.240 | 0.004 | 1.000 | 0.048 | 0.048 | See | See |
| 18 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 2 | 0.004 | 0.000 | 0.359 | 0.0 | -1 | 1 | 3.732 | 0.013 | 0.929 | 0.001 | 0.001 | See | See |
| 21 | KNeighborsClassifier_brute_force | fit | 100000 | 100000 | 2 | 0.004 | 0.000 | 0.367 | 0.0 | -1 | 5 | 0.241 | 0.004 | 1.000 | 0.018 | 0.018 | See | See |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 1.986 | 0.161 | 0.000 | 0.002 | -1 | 1 | 0.312 | 0.002 | 1.000 | 6.373 | 6.374 | See | See |
| 2 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.025 | 0.003 | 0.000 | 0.025 | -1 | 1 | 18.360 | 0.222 | 0.757 | 0.001 | 0.001 | See | See |
| 4 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.795 | 0.034 | 0.000 | 0.003 | -1 | 5 | 18.269 | 0.000 | 0.882 | 0.153 | 0.153 | See | See |
| 5 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.026 | 0.003 | 0.000 | 0.026 | -1 | 5 | 0.318 | 0.006 | 1.000 | 0.081 | 0.081 | See | See |
| 7 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.067 | 0.005 | 0.000 | 0.002 | 1 | 100 | 0.317 | 0.005 | 1.000 | 6.525 | 6.526 | See | See |
| 8 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.022 | 0.001 | 0.000 | 0.022 | 1 | 100 | 18.308 | 0.023 | 0.757 | 0.001 | 0.001 | See | See |
| 10 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.801 | 0.044 | 0.000 | 0.003 | -1 | 100 | 18.350 | 0.295 | 0.663 | 0.153 | 0.153 | See | See |
| 11 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.026 | 0.003 | 0.000 | 0.026 | -1 | 100 | 0.314 | 0.008 | 1.000 | 0.083 | 0.084 | See | See |
| 13 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 2.055 | 0.003 | 0.000 | 0.002 | 1 | 5 | 0.241 | 0.005 | 1.000 | 8.534 | 8.536 | See | See |
| 14 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.022 | 0.000 | 0.000 | 0.022 | 1 | 5 | 3.683 | 0.016 | 0.922 | 0.006 | 0.006 | See | See |
| 16 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 100 | 1.207 | 0.009 | 0.001 | 0.001 | 1 | 1 | 3.731 | 0.014 | 0.929 | 0.324 | 0.324 | See | See |
| 17 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 100 | 0.021 | 0.001 | 0.000 | 0.021 | 1 | 1 | 0.240 | 0.005 | 1.000 | 0.089 | 0.089 | See | See |
| 19 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 1.786 | 0.044 | 0.000 | 0.002 | -1 | 1 | 0.239 | 0.004 | 1.000 | 7.467 | 7.468 | See | See |
| 20 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 2 | 0.009 | 0.005 | 0.000 | 0.009 | -1 | 1 | 3.687 | 0.014 | 0.922 | 0.002 | 0.002 | See | See |
| 22 | KNeighborsClassifier_brute_force | predict | 100000 | 1000 | 2 | 2.656 | 0.025 | 0.000 | 0.003 | -1 | 5 | 3.720 | 0.014 | 0.896 | 0.714 | 0.714 | See | See |
| 23 | KNeighborsClassifier_brute_force | predict | 100000 | 1 | 2 | 0.008 | 0.006 | 0.000 | 0.008 | -1 | 5 | 0.241 | 0.004 | 1.000 | 0.031 | 0.031 | See | See |
KNeighborsClassifier_kd_tree¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: algorithm=kd_tree.
| estimator | function | n_samples_train | n_samples | n_features | accuracy_score_sklearn | accuracy_score_onnx | diff_accuracy_scores | |
|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.929 | 1.000 | 0.071 |
| 2 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 1.000 | 0.946 | 0.054 |
| 4 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.946 | 0.951 | 0.005 |
| 7 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.951 | 1.000 | 0.049 |
| 8 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 1.000 | 0.946 | 0.054 |
| 10 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.951 | 0.929 | 0.022 |
| 13 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.946 | 1.000 | 0.054 |
| 14 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 1.000 | 0.911 | 0.089 |
| 16 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.929 | 0.894 | 0.035 |
| 19 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.891 | 1.000 | 0.109 |
| 20 | KNeighborsClassifier_kd_tree | predict | 1000 | 1 | 2 | 1.000 | 0.911 | 0.089 |
| 22 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.911 | 0.891 | 0.020 |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.261 | 0.061 | 0.025 | 0.0 | -1 | 1 | 122.045 | 0.000 | 0.929 | 0.027 | 0.027 | See | See |
| 3 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.630 | 0.069 | 0.022 | 0.0 | -1 | 5 | 2.640 | 0.237 | 1.000 | 1.375 | 1.380 | See | See |
| 6 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.622 | 0.059 | 0.022 | 0.0 | 1 | 100 | 123.718 | 0.000 | 0.951 | 0.029 | 0.029 | See | See |
| 9 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.636 | 0.073 | 0.022 | 0.0 | -1 | 100 | 2.654 | 0.197 | 1.000 | 1.370 | 1.374 | See | See |
| 12 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.617 | 0.058 | 0.022 | 0.0 | 1 | 5 | 0.048 | 0.019 | 0.891 | 76.053 | 81.944 | See | See |
| 15 | KNeighborsClassifier_kd_tree | fit | 1000000 | 1000000 | 10 | 3.812 | 0.073 | 0.021 | 0.0 | 1 | 1 | 0.005 | 0.000 | 1.000 | 752.340 | 752.477 | See | See |
| 18 | KNeighborsClassifier_kd_tree | fit | 1000 | 1000 | 2 | 0.001 | 0.001 | 0.016 | 0.0 | -1 | 1 | 0.061 | 0.006 | 0.894 | 0.016 | 0.016 | See | See |
| 21 | KNeighborsClassifier_kd_tree | fit | 1000 | 1000 | 2 | 0.001 | 0.000 | 0.030 | 0.0 | -1 | 5 | 0.005 | 0.000 | 1.000 | 0.105 | 0.105 | See | See |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | iteration_throughput | latency | n_jobs | n_neighbors | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.858 | 1.065 | 0.000 | 0.001 | -1 | 1 | 2.633 | 0.244 | 1.000 | 0.326 | 0.327 | See | See |
| 2 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.002 | 0.000 | 0.000 | 0.002 | -1 | 1 | 121.288 | 0.000 | 0.946 | 0.000 | 0.000 | See | See |
| 4 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 1.062 | 0.455 | 0.000 | 0.001 | -1 | 5 | 125.518 | 0.000 | 0.951 | 0.008 | 0.008 | See | See |
| 5 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.003 | 0.001 | 0.000 | 0.003 | -1 | 5 | 2.703 | 0.389 | 1.000 | 0.001 | 0.001 | See | See |
| 7 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 5.643 | 0.611 | 0.000 | 0.006 | 1 | 100 | 2.641 | 0.187 | 1.000 | 2.137 | 2.142 | See | See |
| 8 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.003 | 0.001 | 0.000 | 0.003 | 1 | 100 | 122.653 | 0.000 | 0.946 | 0.000 | 0.000 | See | See |
| 10 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 3.248 | 0.218 | 0.000 | 0.003 | -1 | 100 | 122.767 | 0.000 | 0.929 | 0.026 | 0.026 | See | See |
| 11 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.005 | 0.001 | 0.000 | 0.005 | -1 | 100 | 2.610 | 0.194 | 1.000 | 0.002 | 0.002 | See | See |
| 13 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 1.745 | 0.412 | 0.000 | 0.002 | 1 | 5 | 0.005 | 0.000 | 1.000 | 335.369 | 335.563 | See | See |
| 14 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.002 | 0.000 | 0.000 | 0.002 | 1 | 5 | 0.039 | 0.001 | 0.911 | 0.041 | 0.041 | See | See |
| 16 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1000 | 10 | 0.955 | 0.327 | 0.000 | 0.001 | 1 | 1 | 0.061 | 0.003 | 0.894 | 15.606 | 15.624 | See | See |
| 17 | KNeighborsClassifier_kd_tree | predict | 1000000 | 1 | 10 | 0.001 | 0.000 | 0.000 | 0.001 | 1 | 1 | 0.005 | 0.000 | 1.000 | 0.198 | 0.198 | See | See |
| 19 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.027 | 0.015 | 0.001 | 0.000 | -1 | 1 | 0.005 | 0.000 | 1.000 | 5.255 | 5.257 | See | See |
| 20 | KNeighborsClassifier_kd_tree | predict | 1000 | 1 | 2 | 0.002 | 0.000 | 0.000 | 0.002 | -1 | 1 | 0.038 | 0.001 | 0.911 | 0.062 | 0.062 | See | See |
| 22 | KNeighborsClassifier_kd_tree | predict | 1000 | 1000 | 2 | 0.023 | 0.001 | 0.001 | 0.000 | -1 | 5 | 0.038 | 0.001 | 0.891 | 0.604 | 0.604 | See | See |
| 23 | KNeighborsClassifier_kd_tree | predict | 1000 | 1 | 2 | 0.002 | 0.000 | 0.000 | 0.002 | -1 | 5 | 0.005 | 0.000 | 1.000 | 0.431 | 0.431 | See | See |
HistGradientBoostingClassifier_best¶onnx (1.10.1) vs. scikit-learn (1.0.dev0)
All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.
| estimator | function | n_samples_train | n_samples | n_features | accuracy_score_sklearn | accuracy_score_onnx | diff_accuracy_scores | |
|---|---|---|---|---|---|---|---|---|
| 1 | HistGradientBoostingClassifier_best | predict | 100000 | 1000 | 100 | 0.824 | 1.0 | 0.176 |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | latency | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | HistGradientBoostingClassifier_best | fit | 100000 | 100000 | 100 | 97.191 | 0.0 | 300 | 0.001 | 0.001 | 0.502 | 0.02 | 0.824 | 193.419 | 193.569 | See | See |
| estimator | function | n_samples_train | n_samples | n_features | mean_duration_sklearn | std_duration_sklearn | n_iter | iteration_throughput | latency | mean_duration_onnx | std_duration_onnx | accuracy_score_onnx | speedup | std_speedup | sklearn_profiling | onnx_profiling | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | HistGradientBoostingClassifier_best | predict | 100000 | 1000 | 100 | 0.107 | 0.002 | 300 | 0.007 | 0.0 | 0.416 | 0.007 | 1.0 | 0.258 | 0.258 | See | See |